1. Field of the Invention
The present invention is generally related to a data analysis method, apparatus and article of manufacture and more particularly to an apparatus, article of manufacture and analysis method for determining and extracting trends from nonstationary time varying data.
Although the present invention finds utility in determining and extracting trends from non stationary time varying data sequences such as financial and climatological data, it is understood that the present invention has application to any non stationary time varying data representative of real world phenomenon. For example, the real world phenomenon to which the invention finds utility may include any of a wide variety of real world phenomena such as population growth, traffic flow and, non stationary time varying data representative of processes including electrical, mechanical, biological, chemical, optical, geophysical or other process(es) that may be analyzed and thereby more fully understood by applying the invention thereto.
2. Background Description
Very often when large volumes of data are collected about real word processes, one of the first components analyzed is an apparently aperiodic component or, known as the data trend, e.g., “the markets are trending up.” For a rigorous analysis of climatologic, financial, electric power consumption and/or inventory change data for example, identifying the trends is very important. Under some circumstances non-periodic terms (i.e., the trends) may overwhelm the result, e.g., in computing the correlation function and spectral analysis, for example. In these instances it is equally important to remove the trend or detrend the data before arriving at meaningful spectral results. Presently, because there is no precise mathematical definition for the aperiodic trend, trends are determined on a totally ad hoc basis.
Typically, the trend is selected as the result of a moving mean, a regression analysis, a filtered operation or simple curve fitting with an a priori functional form. These trend approximations are all subjectively determined based on certain idealized assumptions. Furthermore, normally, the data is not detrended using the same trend approximation. Instead, a typical trend determining method using a simple linear function or straight line base is selected to define a detrend zero reference and the data is rezeroed by removing the reference. Since especially for non-linear and non stationary data, the source of the particular trend is the very same mechanisms that generated the data, neither the trend approximation nor detrended data is particularly accurate. Consequently, approximating the trend using a linear fitting or with a moving mean makes little sense where the underlying mechanism is certainly nonlinear, non stationary and may lack a definitive time scale, e.g., for real world phenomena such as data from climatologic studies and financial data.
Similarly, typical regression analysis and/or filtering are based on stationarity and linearity assumptions that may fit the data well, even for nonlinear regressions. However on a more fundamental level, a fortuitous regression analysis and/or filtering fit for particular data, does not justify applying a particular regression formula globally as a time independent function. Thus, these various prior art curve fitting approaches use an a priori determined functional form that does not necessarily match the trend to the same underlying mechanisms, i.e., those that are inherent in the data. For example, financial data analysis pioneers, R. F. Engle and C. W. J. Granger produced market prediction models that are useful for a special class of non stationary processes. Engle and Granger treated the financial market as a special Auto-Regressive-Integrated-Moving-Average (ARIMA) process that is controlled by a series shocks and relaxations. Of course, as Engle and Granger have acknowledged, not all non stationary data satisfies their special assumptions. So, regardless of the fit, it is very likely that the selected trend has a simplistic functional form that may not support the underlying mechanisms. Unfortunately, suitable analysis methods are not available for the vast majority of data from non stationary and nonlinear signal.
Thus, there is a need for a general method for identifying and determining trends from non stationary data and further, for detrending and determining the variability of such non stationary data. There is especially a need for a way to determine the trend and to detrend data from non stationary and nonlinear processes that do not rely on extrinsic functional or simplifying assumptions.